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PATTERN-MATCHING PROCESSING METHOD AND 
IMAGE PROCESSING APPARATUS 



BACKGROUND OF THE INVENTION 

1. Field of the Invention 

[0001] The present invention relates to a pattern- 
matching processing method and an image processing 
apparatus. More particularly, the present invention 
relates to a pattern-matching processing method and an 
image processing apparatus that relates to a distance 
measuring system adopting image processing, and in which 
pattern matching is performed by generating left and 
right matching images to which pixel interpolation has 
been introduced from the photographed original images of 
an object taken by a stereo camera . 

2 . Description of the Related Art 

[0002] Generally, the so-called stereoscopic distance 
measuring method is widely known as a three-dimensional 
measuring technique adopting image processing, in which 
correlation of a pair of left and right images of an 
object photographed from different positions by a stereo 
camera composed of two cameras is found and the distance 
is found from the parallax of the same object using image 
processing based on the principle of the triangulation 
using the camera parameters such as the position of the 
stereo camera and the focal length. 

[0003] Such a distance measuring method is applied to, 
for example, distance measurement of an object located 
ahead of a vehicle such as a car. A stereo camera facing 
ahead is mounted in a vehicle and the distance to an 
object located ahead of the vehicle is measured based on 
the photographed stereoscopic images of the object ahead 
of the vehicle. 

[0004] The distance information of an object 
calculated based on left and right images photographed by 
a stereo camera is sent to, for example, a vehicle 
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control unit or a warning device, and is used to, for 
example, immediately stop the vehicle according to the 
measured distance or to inform the driver of the distance 
to the vehicle in the road ahead. 

[0005] When such a stereoscopic distance measuring 
system is mounted in a vehicle, the system is usually 
equipped with a pattern-matching processing section, a 
distance measuring section, and an object recognition 
processing section for left and right images taken by a 
stereo camera. 

[0006] In the pattern-matching processing section, 
left and right pattern-matching images in a small area 
are generated from the left and right original images 
taken by the stereo camera, and stereoscopic pattern 
matching is performed to find correlation between each 
other based on the left and right pattern-matching 
images . 

[0007] Based on the matching results obtained by the 
stereoscopic pattern-matching processing, the difference 
between positions of pixels in the left and right 
original images (that is, the parallax) relevant to the 
object photographed is measured, and the distance from 
the stereo camera to the object is calculated using the 
measured parallax and the parameters relating to the 
camera. 

[0008] Based on the generated distance data, whether 
the object photographed in the original image is, for 
example, a vehicle in the road ahead is judged. The 
result of the judgment is sent to a vehicle control unit 
that performs the vehicle traveling control, and so on, 
in order to support driving or to avoid danger. 
[0009] By the way, the pattern-matching images 
generated in the pattern-matching processing section are 
composed of, for example, nine pixels, that is, a 3 x 3 
pattern consisting of three transverse pixels and three 
vertical pixels, which are cut out respectively from the 
left and right original images photographed by the stereo 



camera. 

[0010] When pattern matching is performed on these 
pattern-matching images , if it is assumed that the left 
original image is, for example, the reference image and 
the right original image is a comparison image, the right 
pattern-matching images, which are cut out from the right 
original image while being shifted pixel by pixel, are 
compared with the left pattern-matching image cut out 
from the left original image. At this time, a correlation 
coefficient of each pattern-matching image is calculated 
and the degree of matching is judged according to the 
magnitude of the correlation coefficient. 
[0011] When it is judged that the left pattern- 
matching image and the right pattern-matching image match 
each other, the position of the central pixel in the left 
pattern-matching image of the left original image and the 
position of the central pixel in the right pattern- 
matching image of the right original image are found. The 
parallax in the transverse direction is found from the 
difference between the left pixel position and the right 
pixel position. In addition, the parallax in the vertical 
direction is also found in the same technique. If the 
parallax of the object viewed from the stereo camera can 
be found, it is possible to calculate the distance 
between the stereo camera and the object by reference to 
the parameters of the camera. 

[0012] According to a technique using a technology in 
which , as described above, the three-dimensional position 
of an object ahead of a vehicle is measured, a matching 
position at which left and right pattern-matching images 
match each other is found by performing stereoscopic 
matching between the left and right pattern-matching 
images in small areas extracted from the left and right 
original images photographed in stereo, and the distance 
between the positions of pixels of the corresponding left 
and right pattern-matching images is output as a 
parallax. 
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[0013] As the parallax obtained by processing the 
stereoscopic images are expressed in the unit of pixels 
in the image, if a distance is calculated from the 
parallax expressed in the unit of pixels by using the 
triangulation, the resolution is degraded for longer 
distances to an object. 

[0014] Because of the degradation of resolution, an 
error is produced in the calculated distance. This error 
is ascribed to the fact that the parallax detected from 
images photographed by a stereo camera is expressed in 
the unit of pixels and is also a main cause of the 
degradation of resolution in distance measurement based 
on the principle of the triangulation. This error 
commonly occurs in a case where other matching positions 
in images are found and may affect the judgment of an 
object in the object recognition section significantly. 
[0015] Conventionally, as a consequence, resolution is 
improved to secure the accuracy in distance measurement 
by interpolating pixels in each left original image and 
right original image photographed in stereo, 
respectively, when pattern-matching processing is 
performed. 

[0016] There are various known methods as a 
conventional pixel interpolation techniques. In one of 
them, pixel interpolation is performed based on eight 
pixels located adjacently to the pixel at the position to 
be interpolated, in this method, the pixel values of each 
of eight pixels adjacently surrounding the position to be 
interpolated are averaged and the resultant value is 
taken as a pixel value for the position to be 
interpolated. Then, for the whole of the original image, 
pixel interpolation is performed between two adjacent 
pixels successively. 

[0017] This interpolation technique requires a 
tremendous amount of calculation because interpolated 
images are obtained after the interpolation pixel values 
to be interpolated are calculated for the whole of the 
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original image. As a result, it takes a long time for the 
processing of interpolation; therefore, a more or less 
simplified interpolation technique is also adopted, 
[0018] In the simplified pixel interpolation technique 
for obtaining an interpolated image, two adjacent pixels 
are selected, as basic interpolation processing, and an 
interpolation pixel is interpolated sequentially between 
the two adjacent pixels, with the average of the pixel 
values relating to the two pixels being taken as a pixel 
value for interpolation. 

[0019] First, two transversely adjacent pixels in the 
original image are selected and a pixel is interpolated 
sequentially between the two adjacent pixels, with the 
average of the pixel values of the two pixels being taken 
as a pixel value for interpolation. This process, in 
which a pixel is interpolated between the two 
transversely adjacent pixels in the transverse row in the 
original image, is performed from the top of the original 
image to its bottom sequentially. After this, pixel 
interpolation is performed between two vertically 
adjacent pixels in the original image. By utilizing the 
interpolation pixels already interpolated between pixels 
in the transverse direction as well, pixel interpolation 
with the seven adjacent pixels is performed between the 
two vertically adjacent pixels in the original image. 
According to the pixel interpolation technique described 
above, interpolated images are obtained. 
[0020] In this simplified interpolation technique 
described above, which has been adopted to obtain 
interpolation pixels, for two vertically adjacent pixels 
in the original image, the average value of the two 
adjacent pixels is taken as the interpolation pixel value 
and, as for the vertical direction, the average value of 
pixel values each of the seven adjacent pixels has is 
taken as the interpolation pixel value, with the already 
calculated interpolation pixels being also used, 
therefore, the amount of calculations required for the 
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pixel interpolation technique is considerably reduced 
compared to the case where the pixel value of the 
position to be interpolated is obtained from the average 
value of eight adjacent pixels. 

[0021] However, a distance measuring apparatus to be 
mounted in a vehicle is required to deliver as accurately 
and speedily as possible the distance measurement data to 
be supplied for driving support and warning and, 
therefore, there remains a problem that the amount of 
calculations is still large and much time is required, if 
the conventional or simplified pixel interpolation 
technique is used without changes made thereto for 
generating a pattern-matching image from a stereoscopic 
image. 

[0022] The object of the present invention is, 
therefore, to provide a pattern-matching processing 
method and an image processing apparatus, in which 
pattern matching is performed by introducing pixel 
interpolation to the left and right pattern-matching 
images in small areas extracted from the photographed 
original images of an object taken by a stereo camera. 

SUMMARY OF THE INVENTION 
[0023] In order to solve the above-mentioned problem, 
the pattern-matching processing method according to the 
present invention comprises an area generating step for 
generating left and right areas that define a fixed 
range, respectively, from the left and right images 
photographed in stereo, a pixel generating step for 
generating an interpolation pixel between pixels 
contained in the left or right area, and a pattern- 
matching step for performing pattern matching on the left 
and right areas. 

[0024] In the pattern-matching step, pattern matching 
is performed based on the left and right areas containing 
the interpolation pixels, or pattern matching is 
performed based on the left area and the right area that 
has been interpolated with the interpolation pixels. 



- 7 - 



[0025] Moreover, in the pixel generating step, pixel 
interpolation is performed between two vertically 
adjacent pixels relating to the position to be 
interpolated, after pixel interpolation is performed on 
the left area and the right area that contain at the 
center the pixel, the matching position of which has been 
specified by pattern matching based on the left area and 
the right area, pixel interpolation is performed on the 
right area that contains at the center the pixel, the 
matching position of which has been specified by pattern 
matching based on the left area and the right area, or 
pixel interpolation is performed between two transversely 
adjacent pixels in the left area or the right area. 
[0026] In addition, in the pixel generating step, 
after pixel interpolation is performed between two 
transversely adjacent pixels in the left area or the 
right area, pixel interpolation is performed between two 
vertically adjacent pixels at the position to be 
interpolated based on the average value of plural pixels 
surrounding the position to be interpolated, and when the 
average value is calculated, the plural pixels 
surrounding the position to be interpolated are weighted. 
[0027] Then, in the pixel generating step, the average 
value is calculated by giving weight of a value less than 
1 to the already interpolated pixels among the plurality 
of pixels surrounding the position to be interpolated, 
and pixel interpolation is performed starting from the 
pixel position at which the number of pixels surrounding 
the position to be interpolated is largest, which is the 
target for which the average value is calculated. 
[0028] Moreover, in the pixel generating step, pixel 
interpolation is performed based on two pixels adjacent 
to each other only in the transverse direction in the 
left area and the right area. 

[0029] The image processing apparatus according to the 
present invention is a stereoscopic distance measuring 
system that measures the distance to an object that is 
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photographed in images by performing pattern-matching 
processing based on left and right images of the object 
photographed by a stereo camera, to which the above- 
described pattern-matching processing method has been 
applied. 

[0030] According to the image processing apparatus of 
the present invention, the distance to an object ahead of 
a vehicle can be accurately detected, and the period of 
time required for pattern-matching processing for 
detection can be reduced and the distance information for 
driving support and warning can be provided both 
accurately and speedily. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0031] These and other objects and features of the 
present invention will become clear from the following 
description of the preferred embodiments given with 
reference to the attached drawings, wherein: 

FIG.1A and FIG. IB are diagrams that illustrate 
examples of pattern matching of images interpolated with 
pixels according to a first embodiment of the present 
invention . 

FIG.2A and FIG.2B are diagrams that illustrate 
examples of pattern matching of images interpolated with 
pixels according to a second embodiment. 

FIG.3A and FIG.3B are diagrams that illustrate 
examples of pattern matching of images interpolated with 
pixels according to a third embodiment. 

FIG.4A and FIG.4B are diagrams that illustrate 
examples of pattern matching of images interpolated 
pixels according to a fourth embodiment. 

FIG. 5 is a diagram that illustrates an example of 
generating a pattern-matching image by using a first 
pixel interpolation technique according to the present 
embodiments . 

FIG. 6 is a diagram that illustrates an example of 
generating a pattern-matching image by using a second 
pixel interpolation technique according to the present 
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embodiments . 

FIG. 7 is a diagram that illustrates an example of 
generating a pattern-matching image by using a third and 
a fourth pixel interpolation techniques according to the 
present embodiments . 

FIG. 8 is a diagram that illustrates an example of 
generating a pattern-matching image by using a fifth 
pixel interpolation technique according to the present 
embodiments . 

FIG. 9 is a diagram that illustrates a general block 
configuration of an image processing apparatus in a 
stereoscopic distance measuring system using left and 
right images. 

FIG. 10A and FIG. 10B are diagrams that illustrate 
examples of left and right images used for pattern 
matching. 

FIG.11A and FIG. 1 IB are diagrams that illustrate how 
pattern matching is performed using left and right 
images . 

FIG. 12 is a diagram that illustrates pixel 
interpolation in a conventional image processing. 

FIG.13A and FIG.13B are diagrams that illustrate a 
conventional pixel interpolation based on adjacent 
pixels . 

DETAILED DESCRIPTION OF THE INVENTION 
[0032] In order to make the effects of the invention 
clearer, a general process of pattern-matching, relating 
to the present invention, will be explained below. 
[0033] An application example of a stereoscopic 
distance measuring method conventionally used is shown in 
FIG. 9. The example shown schematically is a case where a 
stereo camera facing forward is mounted in a vehicle such 
as a car and the distance to an object ahead of the 
vehicle is measured based on photographed stereoscopic 
images . 

[0034] A stereoscopic distance measuring system 
comprises a stereo camera composed of a set of two 
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cameras, a left camera 1 and a right camera 2, mounted so 
as to face forward, and an image processing apparatus 3. 
[0035] The image processing apparatus 3 comprises a 
pattern-matching unit 31, a distance measuring unit 32 
and an object recognition logic unit 33. FIG. 9 shows only 
the main parts for image processing, but there are 
provided, although not shown here, an image input unit 
that performs input processing on images of an object 
ahead of the vehicle photographed by the left camera 1 
and the right camera 2, an original image memory that 
stores the photographed images processed in the image 
input unit as original images, a distance image memory 
that stores distance-related images generated by 
processing the original images, and so on. 
[0036] The two cameras that constitute the stereo 
camera, that is, the left camera 1 and the right camera 2 
are synchronized with each other and are CCD cameras, the 
shutter speed of which is variable, and one of the CCD 
cameras is used as a main camera that photographs a 
reference image for stereoscopic processing and the other 
is used as a sub camera that photographs a comparison 
image for stereoscopic processing, both having the same 
base line length and being arranged so that the their 
axes perpendicular to the image surface are parallel to 
each other. 

[0037] The image input unit is provided with various 
functional circuits such as an amplifier for processing 
two-system analog photographed image signals from the 
cameras 1 and 2, respectively, and an A/D converter or 
the like for image processing, and moreover, it is 
further provided with an image adjusting circuit for 
electrically correcting slight deviations in the optical 
positions of the cameras 1 and 2. Therefore, in the image 
input unit, images photographed by the cameras 1 and 2 
are converted into digital image data with a fixed 
gradation of luminance and are stored in the original 
image memory after errors in the mechanical mounting 
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positions of each camera 1 and 2 are corrected by image 
adjustment . 

[0038] The pattern-matching unit 31 is composed of 
various circuits and memories for processing such as an 
adder, a dif f erentiator , an absolute value calculating 
circuit, and so on, and generates left and right pattern- 
matching images in small areas from a pair of original 
images, that is, a left image and a right image stored in 
the original image memory, and performs stereoscopic 
pattern-matching to find a mutual correlation according 
to each pattern-matching image. 

[0039] The distance measuring unit 32 measures the 
distance between pixels (= parallax) in the original 
image, which is caused according to the distance to the 
photographed object, based on the matching result 
obtained in the pattern-matching unit 31, and generates 
the data of distance to the object photographed in the 
original image by reference to camera parameters. 
[0040] The object recognition logic unit 33 judges 
whether the object photographed in the original image is, 
for example, the vehicle ahead based on the distance data 
generated in the distance measuring unit 32. The result 
of judgment is sent to a vehicle control device 4 that 
performs vehicle traveling control to support driving and 
avoid danger and, in addition, it is also sent to a 
warning device 5 that warns the driver of the distance to 
the vehicle ahead. 

[0041] Examples of pattern-matching images are shown 
in FIG. 10, which are generated in the pattern-matching 
unit 31 in the image processing apparatus 3 configured as 
described above. In FIG. 10, a pattern-matching image, 
which is cut out from an original image photographed by a 
stereo camera for pattern-matching processing, is 
composed of nine pixels in a 3 x 3 pattern having 3 
columns and 3 rows. FIG. 10A shows the pattern of a 
pattern-matching image cut out from the left original 
image photographed by the left camera 1, and FIG. 1 0B 
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shows the pattern of a pattern-matching image cut out 
from the right original image photographed by the right 
camera 2. In these figures, each pixel corresponds one 
element in a grid. 

[0042] If, for example, the left original image is 
used as a reference image and the right original image, 
as a comparison image, pattern-matching of these pattern- 
matching images is performed by comparing the pattern- 
matching image in FIG.10A, which is cut out from the left 
original image, with the pattern-matching image in 
FIG. 1 OB, which is cut out from the right original image, 
each time being shifted pixel by pixel. Then, the 
correlation coefficient of each pattern-matching image is 
calculated and the degree of matching is judged according 
to the correlation coefficient. 

[0043] FIG. 11 shows how the degree of matching of each 
pattern-matching image is judged. In FIG. 11 also, a left 
original image X L is taken as a reference image and a 
right original image X R , as a comparison image. These 
figures show a case where an object O ahead is 
photographed by a stereo camera; FIG.11A shows the left 
original image X L photographed by the left camera 1 and 
FIG. 1 IB shows the right original image X R photographed by 
the right camera 2. In the left original image X L , an 
object 0 L at the position viewed from the left camera 1 
is shown and in the right original image X R , an object 0 R 
at the position viewed from the right camera 2 is shown, 
respectively. As the object O is located a fixed distance 
ahead of the stereo camera, the positions of the object O 
in the images appear different. 

[0044] Next, how to find the parallax between images 
by pattern-matching is described in an example of a case 
where the right end line of the object 0 is processed. 
[0045] First, it is assumed that a pattern-matching 
image PT L can be extracted from the left original image 
X L , which has a pixel on the right end line of the object 
0 L as its center, for example, p 22 of the 3x3 pattern in 
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FIG.10A. Then, a pattern-matching image PT R , the vertical 
position of which is the same as that of the pattern- 
matching image PT L , is extracted from the right original 
image X R . The next pattern-matching image PT R is 
extracted after the present pattern-matching image PT R 
having a 3 x 3 pattern is shifted each time pixel by 
pixel, for example, to the right in the transverse 
direction, while the vertical position is maintained 
unchanged. 

[0046] Each time the pattern-matching image PT R is 
extracted after being shifted pixel by pixel, the 
correlation coefficient with respect to the pattern- 
matching image PT L is found based on the nine pixel 
values contained in the pattern-matching image PT R . The 
position of the image, which has the least correlation 
coefficient among those obtained by shifting the pattern- 
matching image PT R pixel by pixel, is judged to match the 
pattern-matching image PT L in pattern. 
[0047] When the pattern-matching image X L and the 
pattern-matching image X R are judged to match each other, 
a pixel position P L of the central pixel p 22 in the 
pattern-matching image X L in the left original image X L 
and a pixel position P R of a central pixel q 22 in the 
pattern-matching image X R in the right original image X R 
are found. According to the principle of finding a 
horizontal parallax, it can be found from the difference 
(P L - P R ) between the pixel position P L and the pixel 
position P R . A vertical parallax can be found in the same 
manner. If the parallax of the object O viewed from the 
stereo camera can be found, the distance between the 
stereo camera and the object O can be calculated by 
reference to camera parameters. 

[0048] According to the stereoscopic distance 
measuring method described above, the mutual matching 
position between the left and right pattern-matching 
images is specified by performing a stereoscopic matching 
on the left and right pattern-matching images in the 



- 14 - 



small areas extracted from the left and right original 
images photographed in stereo, and the distance between 
pixels in the corresponding left and right pattern- 
matching images is output as a parallax. 
[0049] However, the unit of the parallax obtained by 
processing stereoscopic images is a pixel in the images. 
Therefore, if a distance is measured by triangulation 
using a parallax expressed in units of pixels, resolution 
is degraded for far objects. 

[0050] The resolution is described by reference to 
FIG.11A and FIG. 1 IB. The position of the right end line 
of the object 0 L in the left original image X L is found 
as the pixel position P L and the position of the right 
end line of the object 0 R in the right original image X R 
is found as the pixel position P R , and it is assumed that 
those positions are expressed by the number of pixels 
located to left thereof in the images as, for example, 
P L =600 and P R =500. Then, the parallax between the images 
is expressed as P L - P R =100. 

[0051] Even though it is found that the distance 
calculated using the triangulation method based on the 
above-mentioned parallax corresponds to 10 m and the 
pixel position P L is 600, there is possibility that the 
position of the right end line of the object O l in the 
left original image X L is actually, for example, 600.5, 
and if this is the case, the parallax should be 100.5. 
However, the distance calculated based on this will be, 
for example, 9.8m, resulting in occurrence of an error. 
Such an error is caused because the unit of the parallax 
obtained from the images photographed by a stereo camera 
is a pixel and, therefore, according to the principle of 
the triangulation, the resolution in distance measurement 
is degraded and the judgment on the object is 
considerably affected. 

[0052] Conventionally, resolution is improved to 
maintain the accuracy in distance measurement by 
performing pixel interpolation on the left original image 
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and the right original image photographed by a stereo 
camera, respectively, when pattern-matching processing is 
performed in the pattern-matching unit 31. The 
conventional pixel interpolation technique is described 
below by reference to FIG. 12 and FIG. 13. 

[0053] FIG. 12 shows a case where pixel interpolation 
is performed based on eight pixels adjacently surrounding 
the position to be interpolated. In this figure, the 
position to be interpolated is denoted by x 5 and eight 
pixels adjacently surrounding the position are denoted by 
x 1 to x 4 , and x 6 to x 9 in the original image. When the 
position x 5 is interpolated with pixels, the pixel value 
of position x 5 is calculated based on the pixel values of 
the eight adjacent pixels and using the following 
expression 

x s = (x x + x 2 + x 3 + x 4 + x 6 + x 7 + x 8 + x 9 ) /8 
Then, position x 5 is interpolated with the value. In some 
cases, each pixel may be weighted. According to this 
interpolation technique, pixel interpolation is performed 
sequentially between two adjacent pixels for the whole of 
the original image. 

[0054] According to this interpolation technique, it 
takes a tremendous amount of time to find the 
interpolation pixel values, and as a result, 
interpolation processing requires a long time, therefore, 
a simplified interpolation technique is adopted, to deal 
with this problem, as shown in FIG. 13. FIG.13A shows an 
example of an original image, but in this case it is 
assumed that the original image has 5x5 pixels for 
simplification of description. Pixels x lx to x 55 are 
arranged. FIG.13B shows an example of an interpolated 
image, in which an interpolation pixel has been 
interpolated between individual pixels that make up the 
original image shown in FIG.13A. The interpolated image 
now has a 9 x 9 pixel array, which is the result of the 
interpolated pixels. In FIG.13B, the original pixel to be 
interpolated with pixels is shown in a bold line frame 
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and an interpolation pixel is shown in a thin line frame, 
[0055] The pixel interpolation technique to obtain the 
interpolated image shown in FIG.13B is as follows. First , 
an interpolation pixel is interpolated sequentially 
between two pixels arranged in the transverse direction 
in the original image. Regarding interpolation pixel y xl 
to be interpolated between pixel x xl and pixel x 12/ the 
interpolation pixel value y lx is obtained by the 
following expression 

Yn = (x u + * 12 ) /2 
Then, regarding interpolation pixel y 12 to be 
interpolated between pixel x 12 and pixel x 13/ 
interpolation pixel value y 12 is obtained by the 
following expression 

Yi2 = (x 12 + x 13 ) /2 
In this manner, pixel interpolation is performed 
sequentially between two pixels adjacent to each other in 
the transverse direction in the original image from the 
top end to the bottom end. 

[0056] After pixel interpolation has been performed 
between two pixels adjacent to each other in the 
transverse direction in the original image, pixel 
interpolation is performed between two pixels adjacent to 
each other in the vertical direction in the original 
image. First, regarding interpolation pixel z xl to be 
interpolated between pixel x xl and pixel x 21 , 
interpolation pixel value z 13L is obtained by the 
following expression using already obtained transverse 
interpolation pixels y lx and y 21 adjacent to pixel 
position z xl 

Zn = (Xn + Yn + x 21 + y 21 ) /4 
Then, regarding pixel position z 12 to be interpolated, 
interpolation pixel value z 12 is obtained by the 
following expression using already obtained interpolation 
pixels y 11# y 21 and z xl 

Z12 - (Xn + Yn + x 12 + z xl + x 21 + y 21 + x 22 ) /7 
In this manner, pixel interpolation with seven adjacent 
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pixels is performed between two pixels adjacent to each 
other in the vertical direction in the original image 
using the interpolation pixels already interpolated 
between pixels in the transverse direction. The 
interpolated image shown in FIG.13B is obtained by 
performing pixel interpolation on the original image 
shown in FIG.13A using the pixel interpolation technique 
described above. 

[0057] According to the pixel interpolation technique 
shown in FIG. 12, the pixel value of the position to be 
interpolated is obtained from the average value of the 
eight adjacent pixels. Compared to this pixel 
interpolation technique, according to the pixel 
interpolation technique adopted in order to obtain the 
interpolated image shown in FIG.13B, the average value of 
two adjacent pixels is taken as an interpolation pixel 
value to be interpolated between the two pixels adjacent 
to each other in the transverse direction in the original 
image, and the average value of seven adjacent pixels 
including the already calculated interpolation pixels is 
taken as an interpolation pixel in the vertical 
direction, therefore, the amount of calculations required 
for the pixel interpolation technique shown in FIG.13B is 
small . 

[0058] However, a distance measuring device to be 
mounted in a vehicle is required to be able to provide 
distance information for driving support and warning with 
high precision in measurement and also able to calculate 
as speedily as possible. When pattern-matching images are 
generated from stereoscopic images, if the conventional 
pixel interpolation technique is used without any changes 
made to it, there still remains a problem that the amount 
of calculations is large and much time is required. 
[0059] According to the pattern-matching processing 
method and the image processing apparatus relating to the 
present invention, the period of time required for 
performing pattern-matching can be reduced by improving 
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the pixel interpolation technique for the left and right 
images for pattern-matching in small areas extracted from 
the photographed original image of an object taken by a 
stereo camera, and by reducing the amount of time 
required for generating pattern-matching images. 
[0060] Next, the embodiments of the pattern-matching 
processing system and the image processing apparatus 
according to the present invention are described below by 
reference to FIG.l to FIG. 8, using an example case where 
a pattern-matching processing is adopted, into which 
pixel interpolation for the photographed original images 
of an object taken by a stereo camera is introduced, in a 
stereoscopic distance measuring system. 
[0061] In a conventional stereoscopic distance 
measuring system, prior to pattern-matching processing, 
left and right original images photographed by a stereo 
camera are turned into interpolated images with doubled 
resolution by performing pixel interpolation on the whole 
of the original images, and pattern matching is performed 
based on the left and right interpolated images. This 
pattern-matching processing requires a tremendous amount 
of calculations and time for generating interpolated 
images . 

[0062] According to the pattern-matching processing in 
the first embodiment, which is applied to the 
stereoscopic distance measuring system relating to the 
present invention, the amount of calculations and the 
period of time required for generating interpolated 
images are reduced while the accuracy in distance is 
improved by generating pattern-matching images that have 
been interpolated with pixels and by performing pattern 
matching on the left and right pattern-matching images 
only when pattern matching is performed. 

[0063] In the first embodiment, when pattern-matching 
processing is performed based on left and right original 
images photographed by a stereo camera, pixel 
interpolation is performed on small areas cut out for 
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pattern matching, for example, patterns of 3 x 3 f that 
is, nine pixels, as shown in FIG. 10A and FIG.10B. 
[0064] FIG.l shows examples of pattern-matching images 
after pixel interpolation has been performed on patterns 
of 3 x 3, that is, nine pixels, cut out from left and 
right original images. FIG.1A shows a state in which 
pixel interpolation has been performed on nine pixels p lx 
to p 33 in FIG.10A, and FIG. IB shows a state in which 
pixel interpolation has been performed on nine pixels q xl 
to q 33 , respectively. In the figures, the pixels that 
have been interpolated with pixels are shown in bold line 
frames . 

[0065] The conventional pixel interpolation technique 
shown in FIG.13B is adopted for generating the pattern- 
matching images shown in FIG.l. First, the reference 
image for pattern-matching processing is generated by 
performing pixel interpolation between two transversely 
adjacent pixels, when nine pixels p ix to p 33 are specified 
as a small area pattern shown in FIG.10A. As a result, 
six interpolation pixels a n to a 32 are obtained. Next, 
from the average value of seven pixels based on nine 
pixels p u to p 33 and the already obtained six 
interpolation pixels a xl to a 32 , 10 interpolation pixels 
b lx to b 25 in the vertical direction are obtained, and the 
left pattern-matching image shown in FIG.1A is thus 
generated. 

[0066] Moreover, regarding nine pixels q lx to q 33 shown 
in FIG. IB making up the comparison image, six 
interpolation pixels c xl to c 32 in the transverse 
direction and 10 interpolation pixels d xl to d 25 in the 
vertical direction are obtained in the same pixel 
interpolation technique, and the right pattern-matching 
image shown in FIG. IB is generated. 

[0067] After the left pattern-matching image and the 
right pattern-matching image are thus generated, pattern 
matching is performed on the 5x5 pixel patterns based 
on these images. In pattern matching, for example, pixels 
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Pn/ b xli p 21 , b 21 , p 31 , . .., in the left pattern-matching 
image are compared with pixels q llf d 11# q 21 , d 21/ q 31/ . 
in the right pattern-matching image, and each difference 
between corresponding pixel values is obtained, 
respectively. In this manner, each difference between 
corresponding pixel values is obtained for the whole of 
the 5x5 pixel patterns, and a correlation coefficient 
between the left and right pattern-matching images is 
found. 

[0068] Next, three vertically adjacent pixels, which 
will be obtained by shifting the pattern by one pixel in 
the transverse direction, are cut out from the right 
original image and, based on these three vertically 
adjacent pixels and original pixels q 13 , q 23 and q 33 , 
interpolation pixels c 13 , d 16 , c 23 , d 26 and c 33 , though not 
shown here, are added to the right end of the right 
pattern-matching image shown in FIG. IB. In this manner, a 
right pattern-matching image, which is composed of 5 x 5 
pixel pattern containing the newly interpolated pixels, 
that is, c xl to c 33 , and which is shifted by one pixel 
from the previous image, is generated as the next 
comparison image. Then, pattern matching is performed 
between the left pattern-matching image shown in FIG.1A, 
which is the reference image, and the right pattern- 
matching image newly generated, which is the next 
comparison image. 

[0069] A new right pattern-matching image of 5 x 5 
pixel pattern, which will be the next comparison image, 
is generated by performing pixel interpolation each time 
so that a 5 x 5 pixel pattern shifted by one pixel is 
generated sequentially in accordance with the 3x3 pixel 
pattern. Then, it is possible to judge that the right 
pattern-matching image, which has the least correlation 
coefficient in comparison between each right pattern- 
matching image and the left pattern-matching image, is 
best matched in the transverse direction and to specify 
the transversely matching position in the original image. 
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In addition, regarding the vertical direction, if the 
right pattern-matching image of 5 x 5 pixel pattern, 
which includes interpolation pixels obtained by shifting 
the right pattern-matching image, which is the comparison 
image, by one pixel in the vertical direction, is 
generated sequentially, the vertically matching position 
can be specified. 

[0070] in the pattern matching processing according to 
the first embodiment, as described above, the right 
pattern-matching image of 5 x 5 pixel pattern is 
generated sequentially by pixel interpolation during 
processing, therefore, the amount of calculations and the 
period of time required for pattern-matching processing 
can be reduced, with doubled resolution being achieved, 
compared to the case where the original image is just 
interpolated with pixels to achieve doubled resolution. 
[0071] Although both the left and the right pattern- 
matching images shown in FIG.1A and FIG. IB are generated 
for the pattern-matching processing of the left and right 
5x5 pixel patterns interpolated with pixels, according 
to the pattern-matching processing in the second 
embodiment, a 3 x 3 pixel pattern cut out from the left 
original image is used, without any changes being made to 
it, as the left pattern-matching image, that is, the 
reference image, and a 3 x 3 pixel pattern cut out from 
the right original image is interpolated with pixels so 
that a right pattern-matching image of 5 x 5 pixel 
pattern, which will be a comparison image, is generated. 
FIG. 2 A shows the left pattern-matching image, which is 
the 3x3 pixel pattern cut out from the left original 
image without any changes made to it, and FIG.2B shows 
the right pattern-matching image, which is a 5 x 5 pixel 
pattern interpolated with pixels. The pixel interpolation 
technique for generating the right pattern-matching image 
is the same as that shown in FIG.1A and FIG. IB. 
[0072] In the pattern matching using the left and 
right pattern-matching images shown in FIG.2A and FIG.2B, 
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3x3, that is, nine pixels are used in the pattern- 
matching processing. Pixels p 11# p 21 , p 31 , ♦ P32 in the 
left pattern-matching image are compared with pixels q llf 
q 2 i/ q 3 i# ^33 in the right pattern-matching image. By 

this comparison, correlation coefficients between the 
left and right pattern-matching images are found. 
[0073] Next, interpolation pixels c 13 , d 16 , c 23 , d 26 and 
c 33 are added in the same manner as that shown in FIG.1A 
and FIG. IB for generating a right patter-matching image, 
which will be the next comparison image. In this manner, 
a right pattern-matching image, which is a 5 x 5 pixel 
pattern shifted by one pixel and includes the newly added 
interpolation pixels, that is, a pixel pattern of pixels 
c n to c 3 3 / is generated as the next comparison image. 
Then, nine pixels p llf p 21 , p 31 , ♦••/ P 32 i n the left 
pattern-matching image shown in FIG.2A, which serves as 
the reference image, are compared with nine pixels c llf 
c 2 i/ c 31 , c 13 , c 23 , c 33 in the right pattern-matching 

image, which has been newly generated and serves as the 
next comparison image. As a result, correlation 
coefficients can be found. 

[0074] While a 3 x 3 pixel pattern of the left 
original image is used without any changes made to it as 
the left pattern-matching image serving as the reference 
image, regarding the right pattern-matching image, which 
serves as a comparison image, pixel interpolation is 
performed each time to generate the right pattern- 
matching image, which is a 5 x 5 pixel pattern serving as 
the next comparison image, so that a 5 x 5 pixel pattern 
shifted by one pixel is generated sequentially. Then, by 
the comparison between each right pattern-matching image 
and the left pattern-matching image, the right pattern- 
matching pattern image with the least correlation 
coefficient is judged to be matched in the transverse 
direction and the transversely matching position in the 
original image can be specified. 

[0075] Regarding the vertical direction, the right 
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pattern-matching image serving as a comparison image is 
shifted by one pixel in the vertical direction and the 
right pattern-matching image of 5 x 5 pixel pattern is 
generated sequentially as a result by using interpolation 
pixels d n to d 25 . In this manner, the vertically matching 
position can be specified. 

[0076] in the second embodiment, as described above, 
when pattern matching is performed, pixel interpolation 
is performed only for generating the right pattern- 
matching image serving as a comparison image and the 
right pattern-matching image of 5 x 5 pixel pattern is 
generated sequentially, therefore, it is possible to 
further reduce the amount of calculations and the period 
of time required for pattern-matching processing while 
achieving doubled resolution, because pixel interpolation 
is not performed for generating the left pattern-matching 
image, compared to the case where pixel interpolation is 
performed so that the original image achieves doubled 
resolution without any changes made to it. 
[0077] In the pattern-matching processing according to 
the first and second embodiments, which have been 
described above by reference to FIG.l and FIG. 2, all of 
the pattern matching between the left and right pattern 
matching images is performed using the images 
interpolated with pixels. However, according to the third 
embodiment, in order to improve the efficiency of 
pattern-matching processing, after or if a mutually 
matching position can be approximately specified using 
normal 3x3 pixel patterns, a pattern-matching image is 
generated by performing pixel interpolation on peripheral 
pixels around the specified position, for example, a 
small area within a pixel distance of the specified 
position . 

[0078] It is assumed that the pixel q 12 is specified 
as the matching position for the right pattern matching 
as a result when a normal pattern-matching processing is 
performed on the left and right 3x3 pixel patterns 
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shown in FIG.10A and FIG . 10B, which have been cut out 
from the left and right original images. However, even 
though it is assumed that the pixel q 12 is specified as 
the matching position, the resolution in matching remains 
unchanged as before and all that can be said is that the 
position has been specified approximately. If the 
position can be specified, the resolution can be 
increased and the efficiency of pattern-matching 
processing can be improved by generating pattern-matching 
images from interpolated images and performing pattern 
matching again only in the vicinity of the position. 
[0079] FIG.3A and FIG.3B show the states in which left 
and right pattern-matching images have been generated, 
respectively, by performing pixel interpolation on small 
areas of 3 x 3 pixel region containing the pixel 
surrounding the position, after the position is specified 
approximately. FIG.3A shows the left pattern-matching 
image and FIG.3B shows the right pattern-matching image , 
respectively. FIGs.3 shows a case where matching is 
performed in the transverse direction. 
[0080] The pixel interpolation technique for 
generating the left and right pattern-matching images 
shown in FIG.3A and FIG.3B, respectively, is the same as 
that in the case shown in FIG.l. The pattern-matching 
images shown in FIG.3A and FIG.3B are generated based on 
the front and rear pixels in the transverse direction in 
accordance with pixel q 12 corresponding to the specified 
position. In the left pattern-matching image shown in 
FIG.3A, the pixel corresponding to the specified position 
is p 22 . 

[0081] In matching between the left and right 
patterns, in the case where the left pattern-matching 
image shown in FIG.3A is taken as the reference image, 
first nine pixels h llf p 21 , b 21 , b 12 , a 21 , b 22 , b 13 , p 22 and 
b 23 , which form the 3x3 pixel pattern of the left 
pattern matching image, are compared with nine pixels 
d u/ q 21 , d 21 , d 12 , c 21 , d 22 , d 13 , q 22 and d 23 , which form a 
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comparison image, that is, the 3x3 pixel pattern of the 
right pattern-matching image, and correlation 
coefficients are found, 

[0082] Next, nine pixels b 11# p 21 , b 21 , b 12 , a 21 , b 22 , b 13 , 
p 22 and b 23 , which form the 3x3 pixel pattern of the 
left pattern-matching image are compared with nine pixels 
di 2 / c 21 , d 22 , d 13 , q 22 , d 23 , d 14 , c 22 and d 24 , which form the 
right pattern-matching image, that is, the next 
comparison image, and correlation coefficients are found. 
[0083] The second next comparison image is generated 
from the right pattern-matching image and correlation 
coefficients are found. Then, the next reference image is 
generated from the left pattern-matching image and 
matching is performed with the comparison images 
generated sequentially from the right pattern-matching 
image. In this manner, the matching position is specified 
by the least value among the obtained correlation 
coefficients . 

[0084] As pattern matching is performed using the 
pattern-matching images interpolated with pixels on only 
the vicinity to the specified matching position, which 
has been obtained in a normal pattern matching using the 
3x3 pixel patterns generated from the original image, 
the resolution can be improved without a large amount of 
calculation and time being required. 
[0085] In generating the left and right pattern- 
matching images shown in FIG.3A and FIG.3B according to 
the third embodiment, pixel interpolation is performed on 
both the left pattern-matching image, which is the 
reference image, and the right pattern-matching image, 
which is a comparison image but, in the fourth embodiment 
shown in FIG. 4, the nine-pixel pattern cut out from the 
left original image, which is used in a normal pattern 
matching, is used as the left pattern-matching image, 
that is, the reference image without any changes made to 
it, as shown in FIG.4A, and pixel interpolation is 
performed only on the right pattern-matching image, from 
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which a comparison image is generated, as shown in 
FIG. 4B. 

[0086] In the left and right pattern matching, nine 
pixels p lx , p 21 , p 31 , p 12 , p 22 , p 32 / Pis* P23 and Pasr forming 
the 3x3 pixel pattern of the left pattern-matching 
image, are first compared with nine pixels d llf q 2 i/ d 21 , 
di 2 / c 2i/ d 22/ d i3^ ^22 and d 23 , forming the 3x3 pixel 
pattern of the right pattern-matching image, which is 
used as a comparison image, and correlation coefficients 
are found, 

[0087] Then, while the left pattern-matching image, 
which is used as the reference image, is left as is, it 
is compared with nine pixels d 12 , c 21 , d 22 , d 13 , q 22 , d 23 , 
d 14 , c 22 and d 24 , which form the next comparison image 
shifted by one pixel in the right pattern-matching image, 
and correlation coefficients are found. 
[0088] From the right pattern-matching image, the 
second next comparison image, the third one, and so on, 
are generated sequentially, and correlation coefficients 
are found. The position corresponding to the least value 
among thus obtained correlation coefficients is judged to 
be the matching position. 

[0089] As pattern matching is performed using the 
pattern-matching images interpolated with pixels only for 
generating the comparison images in the vicinity of the 
specified matching position, which has been obtained by a 
normal pattern matching of the 3x3 pixel patterns 
generated from the original image, the amount of 
calculations and the period of time can be further 
reduced and the resolution can also be further improved 
compared to the pattern-matching processing shown in 
FIG. 3. 

[0090] In the pattern-matching processing in the first 
to fourth embodiments described above, what is focused on 
is how to perform pixel interpolation on the pixel 
patterns cut out from the left and right original images 
when the left and right pattern-matching images are 
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generated, and the applied pixel interpolation is 
basically the conventional pixel interpolation technique 
shown in FIG. 10. In the conventional pixel interpolation 
technique, the interpolation pixel value is a pixel value 
calculated by simply averaging the values of the 
peripheral pixels around the position to be interpolated 
with pixels. However, there is a possibility that the 
interpolation pixel value obtained by simply averaging 
the peripheral pixels according to the conventional pixel 
interpolation technique will not reflect the actual case. 
[0091] Below is described the embodiments relating to 
the pixel interpolation technique in which the influence 
of errors due to pixel interpolation is lessened by 
improving the method for averaging peripheral pixels in 
the conventional pixel interpolation method. 
[0092] FIG. 5 shows the case where a left pattern- 
matching image of 5 x 5 pixels is generated by applying a 
first pixel interpolation technique according to the 
present embodiment to the 3x3 pixel pattern cut out 
from the left original image shown in FIG.10A. 
[0093] The manner, in which pixel interpolation is 
performed between two transversely adjacent pixels in the 
3x3 pixel pattern, is the same as the conventional 
pixel interpolation technique, that is, from the average 
value of pixel p xl and pixel p 12 , interpolation pixel a ix 
is obtained, and in the same manner, interpolation pixels 
a 12 to a 32 are obtained sequentially in the transverse 
direction. 

[0094] Then, pixel interpolation is performed between 
two vertically adjacent pixels in the 3x3 pixel 
pattern. In the conventional pixel interpolation 
technique, the interpolation start position to be 
interpolated with pixels is pixel position e 13L at the end 
of, for example, the pixel pattern shown in FIG. 5, but in 
the pixel interpolation technique according to the 
present embodiment, the start position at which pixel 
interpolation is performed between two vertically 
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adjacent pixels is pixel position e 13 . This is because, 
if interpolation is started from pixel position e llf the 
number of pixels for averaging is small and, therefore, 
interpolation is stared from a position at which the 
number of pixels for averaging is large. 
[0095] First, the pixel value to be interpolated to 
the pixel position e 13 is obtained from the following 
expression 

e 13 = (a lx + p 12 + a 12 + a 21 + p 22 + a 22 ). / 6 
Next, the pixel value to be interpolated to pixel 
position e 12 is obtained from the following expression 

e 12 = (Pn + a lx + p 12 + e 13 + p 21 + a 21 + p 22 ) / 7 
Next, the pixel value to be interpolated to pixel 
position e lx is obtained from the following expression 

©11 = (Pll + a il + e !2 + a 21 + P2l) / 5 

Then, the pixel value for pixel position e 14 is obtained. 
In this manner, interpolation pixel values are calculated 
sequentially starting from the position at which the 
number of pixels for averaging is large. The same 
processing is applied to other pixel positions e 21 to e 25 
in the transverse direction. 

[0096] In the pixel interpolation technique according 
to the present embodiment described above, the 
interpolation start position at which pixel interpolation 
is performed between two vertically adjacent pixels is a 
pixel position at which the number of pixels for 
averaging is large. The average value is a simple average 
of plural pixel values, but in a second pixel 
interpolation technique, which is the next embodiment, 
the interpolation pixel values contained in the plural 
target pixel values for averaging are weighted in order 
to lessen the influence of errors due to interpolation 
pixels as significantly as possible. This is designed to 
suppress the influence of errors inherent in 
interpolation pixels because, if the pixels including 
interpolation pixels are used for averaging without any 
changes made to them, it means that these pixels are 
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regarded as the actually existent pixels , as a result. A 
concrete example of a case, where the interpolation pixel 
values are weighted when calculated, is shown in FIG. 6. 
[0097] FIG. 6 shows the case where the 3x3 pixel 
pattern of the left original image shown in FIG. 1 OA is 
interpolated with pixels, and the pixel interpolation 
between two transversely adjacent pixels is the same as 
that used to generate the left pattern-matching image 
shown in FIG. 5. The interpolation procedure for 
performing pixel interpolation between two vertically 
adjacent pixels is the same as that used to generate the 
pattern-matching image shown in FIG. 5. For example, the 
interpolation pixel value of pixel interpolation position 
f 13 is obtained from the following expression 
f 13 = (a n / 2 + p 12 + a 12 / 2 + a 12 / 2 
+ P22 + a 22 / 2) / 4 
where interpolation pixel values a lx , a 12 , a 21 and a 22 are 
assigned with a weight of 1/2. Due to the assignment of 
weight of 1/2, the number of pixels for averaging is 
decreased from six to four. 

[0098] The assignment of weight of 1/2 adopted here is 
only an example. All that is required is the assignment 
of weight should decrease the values of pixels that 
really exist, and therefore the weight should be less 
than one. 

[0099] In the second pixel interpolation technique, as 
described above, the contained interpolation pixel values 
are obtained by assigning a weight to the pixels that 
really exist in averaging for performing pixel 
interpolation between two vertically adjacent pixels and, 
therefore, the reliability of the pattern-matching 
processing of the generated pattern-matching images can 
be improved . 

[0100] In the first and second pixel interpolation 
techniques described above, when pixel interpolation is 
performed between two vertically adjacent pixels, the 
average value of the eight peripheral pixels is used as 
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the interpolation pixel value. Contrary to this, a third 
pixel interpolation technique, which will be described 
below by reference to FIG. 7, adopts a further simplified 
procedure for performing pixel interpolation between two 
adjacent pixels. 

[0101] FIG. 7 shows the case where the 3x3 pixel 
pattern of the left original image shown in FIG.10A is 
interpolated with pixels. The method for obtaining the 
interpolation pixels a lir a 12/ a 21/ a 22 , a 31 and a 32 between 
two transversely adjacent pixels is the same as that used 
to generate the left pattern-matching image shown in 
FIG. 5, but when pixel interpolation is performed between 
two vertically adjacent pixels in the left original 
image, for example, when pixel interpolation is performed 
between pixel p xl and pixel p 21 , interpolation pixel value 
g lx is obtained from the following expression 

9n - (Pn + P21) / 2 
In the same manner, the interpolation pixel values of 

interpolation positions g 13 , g 15 , g 21 , g 23 and g 25 are 

obtained based on the pixels that really exist on the 

upper and lower sides thereof. 

[0102] Next, when the interpolation pixel values of 
interpolation pixel positions h 12 , h 14 , h 22 and h 24 to be 
interpolated with pixels are obtained, the average value 
of the four pixels, which are located on the upper and 
lower sides of and to the left-hand and right-hand sides 
of each interpolation pixel position, is calculated. For 
example, when the interpolation pixel value of pixel 
position h 12 is obtained by calculation using the 
following expression 

h 12 = (a xl + f xl + f 13 + a 21 ) / 4 
The interpolation pixel values of other pixel positions 
can be calculated in the same manner. 
[0103] According to the third pixel interpolation 
technique, as described above, the period of time 
required for pattern-matching processing can be reduced 
compared to the first and second pixel interpolation 
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means, because a technique of averaging four, that is, 
the upper, lower, left and right pixels, is adopted 
instead of a technique of averaging eight peripheral 
pixels for pixel interpolation between two vertically 
adjacent pixels in the original image. 

[0104] Moreover, it is possible to introduce the idea 
of weighted averaging used in the second pixel 
interpolation technique shown in FIG .6 into the third 
pixel interpolation technique for generating the pattern- 
matching image shown in FIG -7 to establish a fourth pixel 
interpolation technique. In the fourth pixel 
interpolation technique, it is designed so that errors 
are avoided in averaging when the interpolation pixel 
values of interpolation pixel positions h 12 , h 14 , h 22 and 
h 24 are obtained by assigning specified weights, because 
the upper, lower, left and right pixels are interpolation 
pixels that do not really exist. 

[0105] For example, the weight is assumed to be 1/2, 
the interpolation pixel value of pixel position h 12 can 
be obtained by the following expression 

h 12 = (a xl /2 + f lx /2 + f 13 /2 + a 21 /2) / 2 
The interpolation pixel values relating to other pixel 
positions can be obtained in the same procedure. 
[0106] In the first to fourth pixel interpolation 
techniques described above, the left and right pattern- 
matching images are generated by performing pixel 
interpolation between two pixels among nine pixels making 
up the 3x3 pixel patterns shown in FIG.10A and FIG. 10B, 
which have been cut out from the left and right original 
images photographed by a stereo camera. 

[0107] However, when the original image photographed 
by a stereo camera is photographed by the interlaced 
scanning method, pixels are arranged in the transverse 
direction at every two vertical pixel positions in the 
original image in a frame. In other words, the next pixel 
row is photographed in the next frame, therefore, the 
period of time required for two frames is necessary to 
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obtain the transverse pixel arrays corresponding to a 
display image of the original image, A fifth pixel 
interpolation technique is one in which pixel 
interpolation is performed using only the transverse 
components, without using the vertical pixel components, 
when detection accuracy is not required in the vertical 
direction in the original image. An example of generation 
of the left pattern-matching image according to the fifth 
pixel interpolation technique is shown in FIG. 8. 
[0108] in the fifth pixel interpolation technique, 
pixel interpolation is performed only between two 
transversely adjacent pixels in one of the original image 
photographed by the interlaced scanning method, 
therefore, six interpolation pixels a 11# a 12 , a 21 , a 22 , a 31 
and a 32 have to be obtained based on pixels p xl to p 33 of 
the 3x3 pixel pattern. 

[0109] For example, interpolation pixel value a xl can 
be obtained using pixel p xl and pixel p 12 from the 
following expression 

a xl = (Pu + P i2 ) / 2 
Other interpolation pixel values can be obtained from the 
same expression using two adjacent pixel values. 
[0110] According to the fifth pixel interpolation 
technique, as described above, when the original image is 
one by the interlaced scanning method, pattern-matching 
images can be generated by a simple method, therefore, 
the processing time can be considerably reduced while the 
detection accuracy in the transverse direction is 
maintained. 

[0111] By applying the first to fifth pixel 
interpolation techniques, described above, to a case 
where left and right pattern-matching images are 
generated from left and right original images 
photographed by a stereo camera, processing time can be 
reduced while the resolution in the pattern-matching 
processing is maintained. 

[0112] According to the present invention, as 
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described above, when pattern matching is performed, a 
pattern-matching processing method is adopted in which 
the original image is interpolated with pixels and 
pattern-matching images are generated, therefore, the 
period of time required for pattern-matching processing 
can be reduced while the resolution of the images is 
maintained, and as a result, it is no longer necessary to 
provide a storage memory to store the pattern-matching 
images, which are the original images interpolated with 
pixels . 

[0113] Moreover, by applying the pattern-matching 
processing method according to the present invention to a 
stereoscopic distance measuring system, the accuracy in 
distance measurement can be improved . 
[0114] In the pattern-matching processing method 
according to the present invention, after the parallax 
position is specified approximately, pattern-matching 
images regarding the position are generated by pixel 
interpolation and then pattern matching is performed, 
therefore, for example, only 100 images of 3 x 3 x 2 have 
to be processed compared to a case where two 640 x 480 
images must be processed in a conventional pixel 
interpolation technique, and as a result, the processing 
time required for replacing the whole of the original 
image with the images interpolated with pixels can be 
reduced. 

[0115] In addition, in the pattern-matching processing 
method according to the present invention, after the 
parallax position is specified approximately, only a 
right pattern-matching image regarding the position is 
generated by pixel interpolation and then pattern 
matching is performed, therefore, for example, only 100 
images of 3 x 3 have to be processed compared to a case 
where a 640 x 480 image has to be processed in a 
conventional pixel interpolation technique, and as a 
result, the processing time required for replacing the 
whole of the original image with the images interpolated 
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with pixels can be reduced. 

[0116] Moreover, in the pattern-matching processing 
method according to the present invention, when pixel 
interpolation is performed on the pixel patterns cut out 
from the original images and pattern-matching images are 
generated, interpolation pixel values are obtained by 
weighting the peripheral pixels and averaging them, 
therefore, errors due to pixel interpolation can be 
lessened. 

[0117] In the pattern-matching processing method 
according to the present invention, when pixel 
interpolation is performed on the pixel patterns cut out 
from the original images and pattern-matching images are 
generated, interpolation pixel values are obtained as an 
average value calculated from only the upper, lower, left 
and right pixels that really exist, therefore, the range 
in which pixel interpolation can be performed is widened 
and an accurate pixel interpolation can be performed. 
[0118] Particularly, when the image photographed by a 
stereo camera is one by the interlaced scanning method, 
pixel interpolation is performed based on the pixels that 
really exist on the left and right sides and pattern- 
matching images are generated, therefore, an accurate 
pixel interpolation can be realized. 

[0119] By using an image processing apparatus, in 
which the above-mentioned pattern-matching processing 
method according to the present invention has been 
applied to a stereoscopic distance measuring system, the 
distance to an object ahead of the vehicle can be 
detected with accuracy and the pattern-matching 
processing time required for detection can be reduced, 
therefore, it is possible to swiftly provide accurate 
distance information for driving support and warning and 
safe driving of the vehicle is promoted. 

[0120] In this specification, description is made with 
the left-hand side and the right-hand side being clearly 
defined such as a left image, right image, left area, 
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right area, and so on, to give a clear description and 
avoid complexity, but the operations are the same even if 
the left-hand side and the right-hand side are switched, 
and it is clear that such a case is also included in the 
present invention . 



